Click any tag below to further narrow down your results
Links
Martin Alderson walks through what he's learned about speeding up agentic AI workflows, drawing on his web performance background. The core argument is that model inference is often not the biggest bottleneck; tool calls, shell startup, polling loops, and hardware choices frequently matter more.
- Inference has two parts to watch: time to first token (dominated by prefill and provider queueing) and tokens per second. Which one matters depends on whether your agent sends short back-and-forth turns or long outputs, and thinking budget can dwarf both.
- Halving inference time only cuts about 20% off a task if provisioning, tool calls, and suspend/resume stay the same. Tool call traces often reveal slow things like xargs spawning hundreds of thousands of processes or a slow .bashrc (nvm is a common culprit).
- Polling loops with sleep are a red flag; the agent is waiting for something that should push a notification instead. Faster underlying hardware (CPU type, local NVMe versus networked disks) and careful RAM management during suspend/resume also make a measurable difference.
Jev is a new AI model that only outputs structured data instead of human language, making it dramatically faster (70-500ms vs seconds) and enabling real-time applications like playing Doom. The author argues this speed advantage could become a new computational primitive for AI, though he suspects competitors can replicate it using simpler inference tricks on existing models.
- Jev generates all structured output in a single forward pass instead of token-by-token, achieving 70-500ms response times compared to seconds for standard LLMs, fast enough to play real-time video games.
- The speed advantage likely doesn't require novel model architecture—you can achieve similar results by prefilling responses and generating only one constrained token with existing LLMs, suggesting Jev lacks a substantial technical moat.
- Structured output could unlock entirely new use cases beyond chatbots by injecting "100ms worth of dirt-cheap intelligence" at decision points throughout applications, though Jev won't match frontier LLMs in raw capability.